NLP Basic Concepts

Natural Language Processing (NLP), as an important branch of the artificial intelligence field, aims to enable computers to understand and process human language, achieving natural communication between humans and machines. With the rapid development of information technology, text data has become an indispensable part of our daily lives. Advances in NLP technology have provided powerful tools for extracting useful information from massive texts and understanding the deep meaning of language. From early rule-based methods, to later statistical learning methods, and now the widespread application of deep learning technologies, the NLP field has experienced multiple technological innovations. Text representation, as one of the core technologies of NLP, its research and progress play a decisive role in improving the performance of NLP systems.

Welcome to the study of NLP basic concepts. This chapter will introduce the basic concepts of NLP, helping you better understand and review related knowledge of NLP.

1.1 What is NLP

NLP is a technology that enables computers to understand, interpret, and generate human language. It is a very active and important research direction in the field of artificial intelligence. Its core task is to simulate the cognitive and usage processes of human language through computer programs. NLP combines knowledge and technologies from multiple disciplines such as computer science, artificial intelligence, linguistics, and psychology, aiming to break down the barriers between human language and computer language, achieving seamless communication and interaction.

NLP technologies allow computers to perform various complex language processing tasks, such as Chinese word segmentation, subword segmentation, part-of-speech tagging, text classification, entity recognition, relation extraction, text summarization, machine translation, and automatic question answering. These tasks not only require computers to identify and process the surface structure of language but also more importantly, to understand the deep meaning behind the language, including complex factors such as semantics, context, sentiment, and culture.

With the development of modern technologies such as deep learning, NLP has made significant progress. By training on large amounts of data, deep learning models can learn complex patterns and structures of language, achieving performance close to or even exceeding human levels on multiple NLP tasks. However, despite this, NLP still faces many challenges, such as dealing with ambiguity, understanding abstract concepts, and handling metaphors and sarcasm. Researchers are working to solve these problems through more advanced algorithms, larger-scale datasets, and more refined language models to continue advancing NLP technology.

1.2 Development of NLP

The development of NLP has evolved from early rule-based methods, to statistical methods, and now to machine learning and deep learning methods. Each technological revolution has greatly promoted the development of NLP technology, enabling significant achievements in tasks such as machine translation, sentiment analysis, entity recognition, and text summarization. With the continuous enhancement of computing power and the optimization of algorithms, the future of NLP will be brighter, playing a more important role in more fields.

Early Exploration (1940s - 1960s)

The early exploration of NLP began after World War II, when people recognized the importance of automatically translating one language into another. In 1950, Alan Turing proposed the Turing test.

He said that if a machine can become part of a conversation using a typewriter and can completely imitate human beings without any obvious difference, then the machine can be considered as being able to think.

This is a test to determine whether a machine can exhibit intelligent behavior indistinguishable from that of a human. During this period, Noam Chomsky proposed the theory of generative grammar, which had an important impact on understanding how machine translation works. However, the machine translation systems of this period were very simple, mainly relying on dictionary lookups and basic word order rules for translation, and the results were not ideal.

Symbolism and Statistical Methods (1970s - 1990s)

After the 1970s, NLP researchers began exploring new areas, including paradigms based on logic and natural language understanding. During this period, researchers were divided into two camps: symbolism (or rule-based) and statistical methods. Symbolism researchers focused on formal languages and generative grammar, while statistical method researchers focused more on statistical and probabilistic methods. In the 1980s, with the improvement of computing power and the introduction of machine learning algorithms, there was a revolutionary change in the NLP field, and statistical models began to replace complex "hand-written" rules.

Machine Learning and Deep Learning (2000s to Present)

Since the 2000s, with the development of deep learning technology, the NLP field has made significant progress. Deep learning models such as Recurrent Neural Networks (RNN), Long Short-Term Memory networks (LSTM), and attention mechanisms have been widely applied to NLP tasks, achieving remarkable results. In 2013, the introduction of the Word2Vec model opened up a new era of word vector representation, providing more effective text representation methods for NLP tasks. In 2018, the release of the BERT model led a new wave of pre-trained language models, bringing new opportunities and challenges to the development of NLP technology. In recent years, Transformer-based models such as GPT-3, by training large parameter models, can generate high-quality text, and in some cases, can even rival human writing.

1.3 NLP Tasks

In the broad research area of NLP, several core tasks form the foundation of the NLP field, covering aspects from basic text processing to complex semantic understanding and generation. These tasks include, but are not limited to, Chinese word segmentation, subword segmentation, part-of-speech tagging, text classification, entity recognition, relation extraction, text summarization, machine translation, and the development of automatic question-answering systems. Each task has its specific challenges and application scenarios, collectively driving the development of language technology and providing powerful tools for processing and analyzing the growing volume of text data.

1.3.1 Chinese Word Segmentation

Chinese word segmentation (CWS) is a fundamental task in the NLP field. When processing Chinese text, due to the characteristics of the Chinese language, there are no obvious separators between words (such as spaces), so it is not possible to directly determine word boundaries through spaces. Therefore, Chinese word segmentation becomes the first step in processing Chinese text, and its purpose is to split continuous Chinese text into meaningful word sequences.

English input: The cat sits on the mat.
English segmentation output: [The | cat | sits | on | the | mat]
Chinese input: Today's weather is really good, suitable for going out to play.
Chinese segmentation output: ["Today", "weather", "really", "good", ",", "suitable", "go", "out", "to", "play", "."]

Correct segmentation results are crucial for subsequent tasks such as part-of-speech tagging, entity recognition, and syntactic analysis. If the segmentation is inaccurate, it will directly affect the effectiveness of the entire text processing workflow.

Input: The lotus flowers at Yonghe Palace are very beautiful.

Correct segmentation: Yonghe Palace | of | lotus flowers | are | very | beautiful | .
Incorrect segmentation 1: Yong | he | Palace of | lotus flowers | are very | beautiful | . (the place name is fragmented)
Incorrect segmentation 2: Yonghe | Palace | of lotus | flowers are | very beautiful | . (the word boundary is confused)

Correct segmentation results are crucial for subsequent tasks such as part-of-speech tagging, entity recognition, and syntactic analysis. If the segmentation is inaccurate, it will directly affect the effectiveness of the entire text processing workflow.

1.3.2 Subword Segmentation

Subword segmentation is a common text preprocessing technique in the NLP field, aimed at further decomposing words into smaller units, i.e., subwords. Subword segmentation is particularly useful for dealing with vocabulary sparsity issues, where rare words or unseen new words can be understood or generated through known subword units. Subword segmentation is especially important for languages with complex spelling and many compound words (such as German) or in pre-trained language models (such as BERT, GPT series).

There are many methods for subword segmentation, including Byte Pair Encoding (BPE), WordPiece, Unigram, SentencePiece, etc. The basic idea of these methods is to decompose words into smaller, frequently occurring segments, which can be single characters, character combinations, or roots and affixes.

Input: unhappiness

Without subword segmentation: the entire word as a unit, output: "unhappiness"
With subword segmentation (assuming BPE algorithm): the word is split into: "un", "happi", "ness"

In this example, through subword segmentation, the word "unhappiness" is decomposed into three parts: the prefix "un" indicating negation, the root "happi" of "happy" indicating happiness, and the noun suffix "ness" indicating state. Even if the model has never seen the complete word "unhappiness", it can understand its general meaning as "the state of being unhappy" through these known subwords.

1.3.3 Part-of-Speech Tagging

Part-of-Speech Tagging (POS Tagging) is a fundamental task in the NLP field, whose goal is to assign a part-of-speech tag to each word in the text, such as noun, verb, adjective, etc. This process usually relies on a predefined set of part-of-speech tags, such as the common tags in English, including nouns (Noun, N), verbs (Verb, V), adjectives (Adjective, Adj), etc. POS tagging is crucial for understanding sentence structure, performing syntactic analysis, and semantic role labeling for advanced NLP tasks. Through POS tagging, computers can better understand the meaning of the text, and thus perform more complex processing such as information extraction, sentiment analysis, and machine translation.

Assuming we have an English sentence: She is playing the guitar in the park.

The result of part-of-speech tagging is as follows:

• She (Pronoun, PRP)
• is (Verb, VBZ)
• playing (Verb, VBG)
• the (Determiner, DT)
• guitar (Noun, NN)
• in (Preposition, IN)
• the (Determiner, DT)
• park (Noun, NN)
• . (Punctuation, .)

Part-of-speech tagging typically relies on machine learning models such as Hidden Markov Models (HMM), Conditional Random Fields (CRF), or deep learning-based recurrent neural networks RNN and long short-term memory networks LSTM, etc. These models learn from large amounts of annotated data to predict the part-of-speech of each word in a new sentence.

1.3.4 Text Classification

Text classification is a core task in the NLP field, involving automatically assigning given text to one or more predefined categories. This technology is widely applied in various scenarios, including but not limited to sentiment analysis, spam detection, news classification, and topic identification. The key to text classification lies in understanding the meaning and context of the text and mapping it to specific categories based on that.

Assume we have a text classification task, the purpose of which is to classify news articles into one of three categories: "sports," "politics," or "technology."

Text: "The NBA playoffs will start next week, and the Lakers and Warriors will face off in the first round."
Category: "Sports"

Text: "The president announced that tariffs will be increased, triggering international trade disputes."
Category: "Politics"

Text: "Apple released a new Macbook equipped with the latest M3 chip."
Category: "Technology"

The success of a text classification task depends on selecting appropriate feature representations and classification algorithms, as well as having high-quality training data. With the development of deep learning technology, using neural networks for text classification has become a trend, which can capture complex patterns and semantic information in text data, thereby achieving significant performance improvements in many tasks.

1.3.5 Named Entity Recognition

Named Entity Recognition (NER), also known as named entity recognition, is a key task in the NLP field, aiming to automatically identify entities with specific significance in the text and classify them into predefined categories such as names, locations, organizations, dates, times, etc. The entity recognition task is important for information extraction, knowledge graph construction, question-answering systems, and content recommendation, as it helps systems understand the key elements and their attributes in the text.

Assume we have an entity recognition task, the purpose of which is to identify entities such as names, places, and organizations from the text.

Input: Li Lei and Han Meimei are residents of Haidian District, Beijing. They plan to travel to Shanghai on April 7, 2024.

Output: [("Li Lei", "Name"), ("Han Meimei", "Name"), ("Haidian District, Beijing", "Location"), ("April 7, 2024", "Date"), ("Shanghai", "Location")]

Through the entity recognition task, we not only identify entities in the text but also understand their categories, providing important information for a deeper understanding of the text content and context. With the development of NLP technology, the accuracy and efficiency of entity recognition are continuously improving, providing strong support for various NLP applications.

1.3.6 Relation Extraction

Relation extraction is a key task in the NLP field, aiming to identify semantic relationships between entities from the text. These relationships can be causal relationships, ownership relationships, family relationships, geographical relationships, etc. Relation extraction is significant for understanding the content of the text, building knowledge graphs, and enhancing the ability of machines to understand language.

Assume we have the following sentence:

Input: Bill Gates is the founder of Microsoft.

Output: [("Bill Gates", "Founder", "Microsoft")]

In this example, the goal of the relation extraction task is to identify the "founder" relationship between "Bill Gates" and "Microsoft" in the text. Through relation extraction, we can extract useful information from the text, helping computers better understand the content of the text and provide support for subsequent tasks such as knowledge graph construction and question-answering systems.

1.3.7 Text Summarization

Text summarization is an important task in NLP, aiming to generate a concise and accurate summary that summarizes the main content of the original text. Depending on the method of generation, text summarization can be divided into two categories: extractive summarization and abstractive summarization.

• Extractive summarization: Extractive summarization composes a summary by directly selecting key sentences or phrases from the original text. The advantage is that the information in the summary comes entirely from the original text, so the accuracy is relatively high. However, since it is merely a concatenation of sentences from the original text, the generated summary may sometimes be less smooth.
• Abstractive summarization: Unlike extractive summarization, abstractive summarization not only involves selecting text fragments but also requires reorganizing and rewriting these fragments and generating new content. Abstractive summarization is more challenging, as it requires understanding the deep meaning of the text and expressing the same information in a new way. Abstractive summarization usually requires more complex models, such as sequence-to-sequence models with attention mechanisms (Seq2Seq).

Assume we have the following news report:

On May 22, 2021, the China National Space Administration announced that the independently developed Mars probe "Tianwen-1" successfully landed on the surface of Mars. This successful mission marks an important step forward for China in deep space exploration. "Tianwen-1" is equipped with various scientific instruments and will conduct scientific exploration work on the surface of Mars for 90 Martian days, aiming to study the geological structure, climate conditions, and the possibility of the existence of life on Mars.

Extractive summary:

The independently developed Mars probe "Tianwen-1" successfully landed on the surface of Mars, marking an important step forward for China in deep space exploration.

Abstractive summary:

The "Tianwen-1" probe successfully achieved a Mars landing, representing a major advancement in China's space exploration.

The text summarization task has wide applications in information retrieval, news delivery, and report generation. Through automatic summarization, users can quickly obtain the core information of the text, save reading time, and improve information processing efficiency.

1.3.8 Machine Translation

Machine translation (MT) is a core task in the NLP field, referring to the process of automatically translating a natural language (source language) into another natural language (target language) using a computer program. Machine translation not only involves the direct conversion of words, but more importantly, accurately conveys the semantics, style, and cultural background of the source language text, making the translation results natural, accurate, and fluent in the target language, thereby overcoming language barriers and promoting communication and understanding between users of different languages.

Assume we have a Chinese sentence: "The weather is very nice today." We want to translate it into English.

Source language: The weather is very nice today.

Target language: The weather is very nice today.

In this simple example, machine translation can accurately convert the Chinese sentence into English, maintaining the meaning and structure of the original sentence. However, when dealing with longer and more complex texts, the challenges of machine translation will increase accordingly. To improve the quality of machine translation, researchers continue to explore new methods and technologies, such as neural network-based Seq2Seq models, Transformer models, etc., which can learn complex mappings between source and target languages, thereby achieving more accurate and fluent translations.

1.3.9 Automatic Question Answering

Automatic question answering (QA) is a high-level task in the NLP field, aiming to enable computers to understand natural language questions and automatically provide accurate answers based on a given data source. The automatic QA task simulates the ability of humans to understand and answer questions, covering from simple fact queries to complex reasoning and explanations. The development of automatic QA systems involves multiple subtasks of NLP, such as information retrieval, text understanding, knowledge representation, and reasoning.

Automatic QA can be roughly divided into three categories: retrieval-based QA, knowledge-based QA, and community-based QA. Retrieval-based QA retrieves answers from a large amount of text through search engines; knowledge-based QA answers questions through structured knowledge bases; community-based QA relies on user-generated QA data, such as QA communities and forums.

The development and optimization of automatic QA systems is an ongoing process. With the advancement of technology and the improvement of algorithms, these systems have shown significant improvements in accuracy, understanding capabilities, and application scope. By combining different types of data sources and technical methods, automatic QA systems are becoming increasingly intelligent and capable of handling complex and diverse questions.

1.4 The Development of Text Representation

The purpose of text representation is to convert the natural form of human language into a form that computers can process, that is, to digitize text data, allowing computers to effectively analyze and process the text. Text representation is a fundamental and essential task in the field of NLP, directly affecting or even determining the quality and performance of NLP systems.

In NLP, text representation involves converting linguistic units in the text (such as characters, words, phrases, sentences, etc.) and their relationships and structural information into forms that computers can understand and manipulate, such as vectors, matrices, or other data structures. Such representations not only need to retain sufficient semantic information for subsequent NLP tasks, such as text classification, sentiment analysis, and machine translation, but also need to consider computational efficiency and storage efficiency.

The development of text representation has gone through several stages, from early rule-based methods, to statistical learning methods, and now to current deep learning technologies. Text representation technology continues to evolve, providing strong support for the development of NLP.

1.4.1 Word Vectors

The Vector Space Model (VSM) is a fundamental and powerful text representation method in the field of NLP, first proposed by Salton from Harvard University. The vector space model converts text (including words, sentences, paragraphs, or entire documents) into vectors in a high-dimensional space to achieve mathematical representation of the text. In this model, each dimension represents a feature item (e.g., a word, phrase, or short phrase), and the value of each element in the vector represents the weight of that feature item in the text, determined by specific calculation formulas (such as term frequency TF, inverse document frequency TF-IDF, etc.), reflecting the importance of the feature item in the text.

The application of the vector space model is extremely extensive, including but not limited to text similarity calculation, text classification, information retrieval, and other natural language processing tasks. It transforms complex text data into a mathematically computable and analyzable form, making text similarity calculation and pattern recognition possible. In addition, by using matrix operations such as eigenvalue calculation and singular value decomposition (SVD), the text vector representation can be optimized, further improving processing efficiency and effectiveness.

However, the vector space model also has many problems. The most important one is the problem of data sparsity and dimension disaster, because the number of feature items is huge, leading to extremely high vector dimensions, and most elements are zero. In addition, due to the model's assumption of independence between feature items, it ignores the structural information in the text, such as word order and context information, limiting the model's expressiveness. The shortcomings of feature item selection and weight calculation methods are also problems that the vector space model needs to solve.

VSM method word vectors:

# "Yonghe Palace's lotus flowers are very beautiful"
# Vocabulary size: 16384, the sentence contains the words: ["Yonghe Palace", "of", "lotus flowers", "very", "beautiful"] = 5 words

vector = [0, 0, ..., 1, 0, ..., 1, 0, ..., 1, 0, ..., 1, 0, ..., 1, 0, ...]
#                    ↑          ↑          ↑          ↑          ↑
#      16384-dimensional only 5 positions are 1, the remaining 16379 positions are 0
# Actual effective dimensions: only 5 dimensions (non-zero dimensions)
# Sparsity rate: (16384-5)/16384 ≈ 99.97%

The vocabulary is a collection containing all possible words. In the vector space model, each word corresponds to a position in the vocabulary, and through this way, words can be converted into vector representations. For example, if the vocabulary size is 16384, then each word will be represented as a 16384-dimensional vector, where only the position corresponding to the word is 1, and the rest are 0.

To solve these problems, researchers' studies on the vector space model mainly focus on two aspects: one is to improve the feature representation method, such as using graph methods, topic methods, etc., for keyword extraction; the other is to improve and optimize the calculation method of feature item weights, which can be combined with existing methods or propose new calculation methods.

1.4.2 Language Models

The N-gram model is a statistical language model widely used in the field of NLP, applied to many tasks such as speech recognition, handwriting recognition, spelling correction, machine translation, and search engines. The core idea of the N-gram model is based on the Markov assumption, which means that the probability of a word appearing depends only on the previous N-1 words. Here, N represents the number of consecutive words, which can be any positive integer. For example, when N=1, the model is called unigram, considering only the probability of a single word; when N=2, it is called bigram, considering the previous word to estimate the probability of the current word; when N=3, it is called trigram, considering the previous two words to estimate the probability of the third word, and so on for N-gram.

The N-gram model estimates the probability of the entire sentence using the chain rule of conditional probability. Specifically, for a given sentence, the model calculates the conditional probability of each N-gram and multiplies these probabilities to get the probability of the entire sentence. For example, for the sentence "The quick brown fox," as a trigram model, we would calculate $P("brown" | "The", "quick")$, $P("fox" | "quick", "brown")$, and so on, and multiply them.

The advantages of the N-gram model are its simplicity and ease of understanding, and it performs well in many tasks. However, when N is large, data sparsity issues arise. The parameter space of the model increases sharply, and the probability of the same N-gram sequence becomes very low, causing the model to be unable to learn effectively, and the model's generalization ability decreases. In addition, the N-gram model ignores the dependency relationships between words and cannot capture the complex structure and semantic information in the sentence.

Despite its limitations, the N-gram model is still widely used in many NLP tasks due to its simplicity and practicality. In some applications, combining the N-gram model with other techniques (such as deep learning models) can achieve better performance.

1.4.3 Word2Vec

Word2Vec is a popular word embedding (Word Embedding) technique proposed by Tomas Mikolov and others in 2013. It is a neural network-based language model (NNLM) that aims to generate dense vector representations of words by learning the contextual relationships between words. The core idea of the Word2Vec model is to capture the semantic relationships between words by using the context information of words in the text, so that semantically similar or related words are closer in the vector space.

The Word2Vec model mainly has two architectures: Continuous Bag of Words (CBOW), which calculates and outputs the vector representation of the target word based on the word vectors of the words in the context of the target word; and Skip-Gram, which is the opposite of the CBOW model, uses the vector representation of the target word to calculate the word vectors of the context. Practical verification shows that CBOW is suitable for small datasets, while Skip-Gram performs better in large corpora.

Compared to traditional high-dimensional sparse representations (such as One-Hot encoding), Word2Vec generates low-dimensional (usually hundreds of dimensions) dense vectors, which help reduce computational complexity and storage requirements. The Word2Vec model can capture the semantic relationships between words, such as the positions of "king" and "queen" being relatively close in the vector space, because they often appear in similar contexts in large texts. The Word2Vec model can also generalize well to unseen words, as it is learned based on context information rather than a dictionary. However, due to the local context-based nature of the CBOW/Skip-Gram model, it cannot capture long-range dependencies and lacks overall relationships between words, thus performing poorly in some complex semantic tasks.

1.4.4 ELMo

ELMo (Embeddings from Language Models) achieved a leap from polysemy of a word and static word vectors to dynamic word vectors. First, a language model is trained on a large corpus to obtain a word vector model, and then the model is fine-tuned for specific tasks to obtain word vectors more suitable for that task. ELMo was the first to introduce the concept of pre-training into the generation of word vectors, using a bidirectional LSTM structure to capture contextual information of words, generating more rich and accurate word vector representations.

ELMo adopts a typical two-stage process: the first stage is to pre-train a language model; the second stage is to extract the word vectors of the corresponding words from the pre-trained network as new features added to downstream tasks during specific tasks. The training time of the RNN-based LSTM model is long, and feature extraction is the key to optimizing and improving the ELMo model.

The main advantage of the ELMo model is its ability to capture the polysemy and contextual information of words, generating more rich and accurate word vectors, suitable for various NLP tasks. However, the ELMo model also has some problems, such as high model complexity, long training time, and high consumption of computing resources.

References

[1] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean. (2013). Distributed Representations of Words and Phrases and their Compositionality. arXiv preprint arXiv:1310.4546.

[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.

[3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. (2023). Attention Is All You Need. arXiv preprint arXiv:1706.03762.

[4] Malek Hajjem, Chiraz Latiri. (2017). Combining IR and LDA Topic Modeling for Filtering Microblogs. Procedia Computer Science, 112, 761–770. https://doi.org/10.1016/j.procs.2017.08.166.

[5] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer. (2018). Deep contextualized word representations. arXiv preprint arXiv:1802.05365.

[6] Salton, G., Wong, A., Yang, C. S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), 613–620. https://doi.org/10.1145/361219.361220.

[7] Zhao Jingsheng, Song Mengxue, Gao Xiang, et al. Research on Text Representation in Natural Language Processing[J]. Journal of Software, 2022, 33(01): 102-128. DOI: 10.13328/j.cnki.jos.006304.

[8] Report on the Development of Chinese Information Processing (2016) Preface[C]//Report on the Development of Chinese Information Processing (2016). Chinese Information Processing Society;, 2016: 2-3. DOI: 10.26914/c.cnkihy.2016.003326.